PARAMETER-EFFICIENT FINE-TUNING

ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models (2024.naacl-long)

Challenge:	Low-Rank Adaptation (LoRA) adapts large language models by training only a small fraction of parameters, but as the rank of the low-rank matrices increases, LoRA exhibits an unstable “double descent” phenomenon, which delays convergence and impairs generalization by causing instability due to the attraction to sharp local minima.
Approach:	They propose a framework that incorporates Momentum-Guided Perturbation Optimization (MGPO) MGPO stabilizes training dynamics by mitigating double descent phenomenon and guiding weight perturbations using momentum vectors from the optimizer’s state.
Outcome:	The proposed framework improves performance on natural language understanding benchmarks and shows that it improves convergence and generalization.

Copied to clipboard

Zequan Liu, Jiawen Lyn, Wei Zhu, Xing Tian

Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Models (2024.findings-emnlp)

Challenge:	Low-rank adaptation (LoRA) has demonstrated commendable performance as a popular method . however, it is implemented with a fixed intrinsic rank that might not be ideal for downstream tasks.
Approach:	They propose a method that estimates the importance score of each LoRA rank and prunes abundant LoRA ranks to improve performance.
Outcome:	The proposed method outperforms baselines on a variety of tasks with comparable parameters.

Copied to clipboard

Vladimir Araujo, Marie-Francine Moens, Tinne Tuytelaars

Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models (2024.emnlp-main)

Challenge:	Recent work shows that PEFT methods can be competitive with, or even superior to, full fine-tuning of PLMs.
Approach:	They propose a method that isolates the training of new PEFT modules to ensure their task specialization and learns to compose them by training a network of routers that leverages a small memory containing examples of previously seen tasks.
Outcome:	The proposed method improves generalization and performance in two CL setups.

Copied to clipboard

Zihan Wang, Deli Chen, Damai Dai, Runxin Xu, Zhuoshu Li, Yu Wu

RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuning (2024.emnlp-main)

Challenge:	Existing studies on parameter-efficient fine-tuning (PEFT) for dense-architecture LLMs are lacking.
Approach:	They propose an expert-specialized fine-tuning method that tunes the experts most relevant to downstream tasks while freezing the other experts.
Outcome:	The proposed method matches or surpasses full-parameter fine-tuning.

Copied to clipboard

Haoyu Wang, Tianci Liu, Ruirui Li, Monica Cheng, Tuo Zhao, Jing Gao

Obliviate: Neutralizing Task-agnostic Backdoors within the Parameter-efficient Fine-tuning Paradigm (2025.findings-naacl)

Challenge:	Pre-trained language models have strong generalizability, but fine-tuning involves updating all parameters, rendering full fine-uning resource-intensive.
Approach:	They propose a parameter-efficient fine-tuning method that updates all pre-trained parameters during inference.
Outcome:	The proposed method outperforms baseline methods on five benchmarks across 20 datasets.

Copied to clipboard

Jaehan Kim, Minkyoo Song, Seung Ho Na, Seungwon Shin

NLoRA: Nyström-Initiated Low-Rank Adaptation for Large Language Models (2025.findings-emnlp)

Challenge:	Parameter-efficient fine-tuning (PEFT) is a key training strategy for large language models, but its reliance on fewer trainable parameters poses security risks, such as task-agnostic backdoors.
Approach:	They propose a method that amplifies benign neurons within PEFT layers and penalizes the influence of trigger tokens.
Outcome:	The proposed method reduces the success rate of state-of-the-art task-agnostic backdoors by 83.6%.

Copied to clipboard

Chenlu Guo, Yi Chang, Yuan Wu

Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning (2023.acl-short)

Challenge:	Parameter-efficient fine-tuning is essential for adapting large language models (LLMs). However, LoRA suffers from slow convergence and some recent LoRA variants, such as PiSSA, rely on Singular Value Decomposition (SVD) for initialization.
Approach:	They propose to introduce a small intermediate matrix between the low-rank matrices (A) and (B) and propose NyströmLoRA (NLoRA) which leverages Nyström-based initialization for SLoRA to improve its effectiveness and efficiency.
Outcome:	The proposed approach improves on 5 natural language generation tasks and 8 natural language understanding tasks with minimal parameter overhead.

Copied to clipboard

Zhen-Ru Zhang, Chuanqi Tan, Haiyang Xu, Chengyu Wang, Jun Huang, Songfang Huang

MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning (2024.acl-long)

Challenge:	Parameter-efficient fine-tuning only optimizes a few task-specific parameters with frozen pre-trained model.
Approach:	They propose to optimize a prefix vector inserted into Transformer layers to optimize the prefix . they propose to use a gate mechanism to adjust the prefixed to each layer .
Outcome:	The proposed approach improves on the SuperGLUE and NER datasets.

Copied to clipboard

Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten Rijke, Zhumin Chen, Jiahuan Pei

FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation (2026.findings-eacl)

Challenge:	Large language models (LLMs) are the default paradigm for natural language processing (NLP) as the models’ scale and the diversity of tasks increase, fine-tuning becomes infeasible.
Approach:	They propose to freeze original pretrained weights and train a group of mini LoRAs with only a small number of parameters and reduce their rank by 8 times .
Outcome:	The proposed model uses fewer trainable parameters while maintaining a higher rank, thereby offering improved performance potential.

Copied to clipboard

Fatema Siddika, Md Anwar Hossen, Juan Pablo Munoz, Tanya G. Roosta, Anuj Sharma, Ali Jannesari

Parameter-Efficient Fine-Tuning without Introducing New Latency (2023.acl-long)

Challenge:	Representation Fine-Tuning (ReFT) adapts large pre-trained models by updating only a small subset of parameters.
Approach:	They propose a method that uses sparse intervention layers to steer hidden representations directly to capture rich semantic information.
Outcome:	The proposed approach outperforms PEFTs on commonsense reasoning, arithmetic reasoning, and GLUE benchmarks while maintaining a high parameter efficiency.

Copied to clipboard

Baohao Liao, Yan Meng, Christof Monz

SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture (2025.naacl-long)

Challenge:	Parameter-efficient fine-tuning of pre-trained language models has been demonstrated to be effective, but its inherent characteristics limit its performance.
Approach:	They propose to generate a sparse mask in a task-agnostic manner by modifying only a small subset of existing parameters and adding new parameters.
Outcome:	The proposed method surpasses existing methods on the GLUE benchmark by a significant margin.

Copied to clipboard

Jiayi Han, Liang Du, Hongwei Du, Xiangguo Zhou, Yiwen Wu, Yuanfang Zhang, Weibo Zheng, Donghong Han

Green & Sustainable NLP

Challenge:	balancing the training budget, downstream performance, and general capabilities of large language models remains a challenge in many applications.
Approach:	They propose a mixture of expert framework based on Soft LoRA and Identity Mixture . SLIM allows dynamic routing between LoRA adapters and identity layers .
Outcome:	The proposed framework reduces training cost while maintaining general capabilities . it can be open-sourced upon publication.

Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillation (2025.findings-acl)

Copied to clipboard

Shuai Zhao, Xiaobao Wu, Cong-Duy T Nguyen, Yanhao Jia, Meihuizi Jia, Feng Yichao, Anh Tuan Luu

Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation (2024.findings-naacl)

Challenge:	Parameter-efficient fine-tuning (PEFT) can bridge the gap between large language models and downstream tasks, but is vulnerable to malicious attacks.
Approach:	They propose a weak-to-strong unlearning algorithm based on feature alignment knowledge distillation to defend against backdoor attacks . they first train a small-scale language model through full-parameter fine-tuning to serve as the clean teacher model and then guide the large-scale poisoned student model in unlearning the backdoor.
Outcome:	The proposed method can unlearn backdoor features without compromising model performance.

Copied to clipboard

Tong Su, Xin Peng, Sarubi Thillainathan, David Guzmán, Surangika Ranathunga, En-Shiun Lee

Prototype-based HyperAdapter for Sample-Efficient Multi-task Tuning (2023.emnlp-main)

Challenge:	Parameter-efficient fine-tuning (PEFT) methods are important in low-resource language (LRL) Neural Machine Translation (NMT) but their practical effectiveness varies significantly across different languages.
Approach:	They evaluated the performance of 8 parameters-efficient fine-tuning methods with 15 architectures using the SacreBLEU score.
Outcome:	The Houlsby+Inversion adapter outperforms the baseline architectures in both in-domain and out-domain tests and the Houlson+Inverter achieves the best performance overall.

Copied to clipboard

Hao Zhao, Jie Fu, Zhaofeng He

Towards Robust and Generalized Parameter-Efficient Fine-Tuning for Noisy Label Learning (2024.acl-long)

Challenge:	Existing methods for fine-tuning pre-trained language models are limited to low-data regimes and require learning different modules to adapt to diverse tasks.
Approach:	They propose a framework for parameter-efficient fine-tuning that trains modules per task . they use an instance-dense retriever and a prototypical hypernetwork to generate conditional modules .
Outcome:	The proposed framework outperforms existing methods on multi-task learning and few-shot transfer learning.

Copied to clipboard

Yeachan Kim, Junho Kim, SangKeun Lee

Black-Box Tuning of Vision-Language Models with Effective Gradient Approximation (2023.findings-emnlp)

Challenge:	Parameter-efficient fine-tuning (PEFT) has enabled efficient optimization of cumbersome language models in real-world environments.
Approach:	They propose a routing-based PEFT approach that adaptively activates PEFT modules.
Outcome:	The proposed method is more sensitive to noise interference than other methods.

Copied to clipboard

Zixian Guo, Yuxiang Wei, Ming Liu, Zhilong Ji, Jinfeng Bai, Yiwen Guo, Wangmeng Zuo

LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluation (2025.coling-main)

Challenge:	Large vision-language models are often not open-source due to preventing abuse or commercial factors.
Approach:	They propose a method for parameter-efficient fine-tuning to improve model accessibility . large models are often not open-source due to preventing abuse or commercial factors . they propose implementing a lightweight adapter over the output feature of an inaccessible model .
Outcome:	The proposed methods improve on 11 benchmarks and are made publicly available.

Copied to clipboard

Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, Tiejun Zhao, Muyun Yang

Green & Sustainable NLP

Challenge:	Low-Rank Adaptation (LoRA) is currently the most commonly used PEFT method for fine-tuning models with billions of parameters.
Approach:	They propose to use low-rank Adaptation to evaluate LoRA parameter features and then retain LoRA for important layers and the other layers share the same LoRA.
Outcome:	The proposed method achieves comparable performance to full fine-tuning and LoRA while retaining 50% of the LoRA parameters on average.

GuiLoMo: Allocating Experts and Ranks for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors (2025.findings-emnlp)

Copied to clipboard

Xinrong Chen, Hengyuan Zhang, Yingmin Qiu, Xiao Liang, Ziyue Li, Guanyu Wang, Weiping Li, Tong Mo, Hayden Kwok-Hay So, Ngai Wong

Green & Sustainable NLP

Challenge:	Low-Rank Adaptation (LoRA) methods are efficient for a large language model with reduced computational costs.
Approach:	They propose a layer-wise expert numbers and ranks allocation strategy with GuidedSelection Vectors.
Outcome:	The proposed method achieves superior or comparable performance to all baselines on three backbone models.

PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learning (2024.findings-acl)

Copied to clipboard

Zhisheng Lin, Han Fu, Chenghao Liu, Zhuo Li, Jianling Sun

Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning (2024.findings-acl)

Challenge:	Parameter-efficient fine-tuning (PEFT) is an effective method for adapting pre-trained language models to various tasks efficiently.
Approach:	They propose a parameter-efficient fine-tuning framework that captures transferable knowledge as a weighted combination of adapters trained on source tasks.
Outcome:	The proposed method yields stable improvements over full fine-tuning and knowledge transferring methods on a broad range of tasks over 17 datasets.

Copied to clipboard

Naibin Gu, Peng Fu, Xiyu Liu, Bowen Shen, Zheng Lin, Weiping Wang

Curriculum Debiasing: Toward Robust Parameter-Efficient Fine-Tuning Against Dataset Biases (2025.acl-long)

Challenge:	Existing methods for fine-tuning large language models are inefficient and redundant . a light-PEFT framework can be used to prune redundant parameters during training .
Approach:	They propose a parameter-efficient fine-tuning framework that freezes most parameters of the foundation model and finetuns only a small number of parameters.
Outcome:	The proposed framework achieves training and inference speedup, reduces memory usage, and maintains comparable performance and plug-and-play feature of PEFT.

Copied to clipboard

Mingyu Lee, Yeachan Kim, Wing-Lam Mok, SangKeun Lee

DisLoRA: Task-specific Low-Rank Adaptation via Orthogonal Basis from Singular Value Decomposition (2025.emnlp-main)

Challenge:	Parameter-efficient fine-tuning (PEFT) addresses the memory footprint issue of full fine- tuning by modifying only a subset of model parameters.
Approach:	They propose a framework that debiases models in a biased-to-unbiased order and uses only a subset of parameters to modify model parameters.
Outcome:	The proposed framework accelerates convergence on unbiased examples by approximately twofold and improves ID and OOD performance by 1.2% and 8.0%, respectively.

Copied to clipboard

She Yifei, Xinhao Wei, Yulong Wang

Reasoning

Challenge:	Extensive experiments on GLUE and Commonsense Reasoning benchmarks demonstrate that DisLoRA surpasses established PEFT methods, including LoRA, PiSSA, DoRA, LoRA-Dash, and SORSA.
Approach:	They propose a framework that leverages singular value decomposition to decompose pretrained weight matrices into orthogonal backbone and task-specific subspaces.
Outcome:	Extensive experiments on GLUE and Commonsense Reasoning benchmarks show that DisLoRA surpasses established PEFT methods, including LoRA, PiSSA, DoRA, LoRA-Dash, and SORSA.

Adapt Once, Thrive with Updates: Transferable Parameter-Efficient Fine-Tuning on Evolving Base Models (2025.acl-long)

Copied to clipboard

Naibin Gu, Peng Fu, Xiyu Liu, Ke Ma, Zheng Lin, Weiping Wang

CoLA: Collaborative Low-Rank Adaptation (2025.findings-acl)

Challenge:	Parameter-efficient fine-tuning (PEFT) is a common method for fine- tuning large language models . however, once updated, PEFT modules suffer performance degradation on newer versions .
Approach:	They propose a method that enhances the PEFT module by focusing on the task-specific pattern while reducing its dependence on certain knowledge in the base model.
Outcome:	Experiments show that PEFT modules can maintain performance on updated models without re-tuning . the proposed approach can be used in real-world applications with large model sizes .

Copied to clipboard

Yiyun Zhou, Chang Yao, Jingyuan Chen

An Empirical Study on the Transferability of Transformer Modules in Parameter-efficient Fine-tuning (2022.emnlp-main)

Challenge:	The scaling law of Large Language Models (LLMs) reveals diminishing return on performance as model scale increases.
Approach:	They propose a more flexible LoRA architecture with an efficient initialization scheme . they propose combining three collaborative strategies to enhance performance .
Outcome:	The proposed model outperforms existing methods in low-sample scenarios.

Copied to clipboard

Mohammad AkbarTajari, Sara Rajaee, Mohammad Taher Pilehvar

Weed Out, Then Harvest: Dual Low-Rank Adaptation is an Effective Noisy Label Detector for Noise-Robust Learning (2025.findings-acl)

Challenge:	Parameter-efficient fine-tuning is a computationally expensive process . introducing new parameters to an already-large model can be considered a drawback.
Approach:	They investigate the capability of different transformer modules in transferring knowledge from a pre-trained model to a downstream task.
Outcome:	The proposed methods show that each transformer module is a winning ticket . they show that with only 0.003% updateable parameters, they can show acceptable performance on target tasks.

Copied to clipboard

Bo Yuan, Yulin Chen, Yin Zhang

Robustness in NLP

Challenge:	Experimental results show that PEFT can fine-tune language models without relying on perfectly labeled datasets.
Approach:	They propose a framework that decouples sample selection from model training by introducing clean and noisy LoRA.
Outcome:	The proposed framework decouples sample selection from model training.

Is Modularity Transferable? A Case Study through the Lens of Knowledge Distillation (2024.lrec-main)

Copied to clipboard

Mateusz Klimaszewski, Piotr Andruszkiewicz, Alexandra Birch

TARE: Lightweight Token-Aware Representation Editing for Fine-tuning Transformer-like Models (2026.acl-long)

Challenge:	Existing approaches to modularity are limited to the case of pre-trained modules in a pre-training language model.
Approach:	They propose a method that allows the transfer of pre-trained PEFT modules between incompatible PLMs without any change in the inference complexity.
Outcome:	The proposed method allows the transfer of modules between incompatible PLMs without any change in the inference complexity.

Copied to clipboard

Yulong Wang, Siyu Zhao

FARSS: Fisher-Optimized Adaptive Low-Rank and Singular-Vector Selection for Knowledge-Preserving Fine-Tuning (2026.findings-acl)

Challenge:	Existing PEFT methods can be costly and underfit token-level contexts.
Approach:	They propose a PEFT method that performs fine-grained, token-specific edits with a small additional inference overhead and minimal tuning.
Outcome:	The proposed method outperforms state-of-the-art methods in 8 tasks and GLUE with a minimal tuning overhead and inference overhead.

Copied to clipboard

Renxing Chen, Ziwei Xiang, Peisong Wang, Hongjian Fang, Meng Li, Fanhu Zeng, Yanan Zhu, Peipei Yang, Xu-Yao Zhang, Jian Cheng

Specialization without Sparsity: Efficient and Expressive Split-Path Experts for LLM Fine-Tuning (2026.findings-acl)

Challenge:	Low-rank adaptation methods for large language models have limitations in preserving world knowledge and limiting updates to preserve world knowledge.
Approach:	They propose a Fisher-optimized adaptive low Rank and Singular-VectorSelection framework for knowledge-preserving fine-tuning that allows efficient and task-sensitive updates.
Outcome:	The proposed framework outperforms existing methods for knowledge-preserving fine-tuning.

Copied to clipboard

Ganghao Liu, Qin Zhou, Zhe Wang, Xuehan Lu, Haihua Huang, Yunfei Tong, Heng Tian

Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models (2025.findings-acl)

Challenge:	Parameter-efficient fine-tuning (PEFT) is a low-cost alternative to full fine-timing due to the massive overhead.
Approach:	They propose a Mixture-of-Experts approach that enhances specialization while maintaining low resource overhead.
Outcome:	The proposed approach outperforms or matches state-of-the-art methods on GLUE, GSM8K, MBPP, and a text rewriting task from SmolTalk.

Copied to clipboard

Sajjad Ghiasvand, Yifan Yang, Zhiyu Xue, Mahnoosh Alizadeh, Zheng Zhang, Ramtin Pedarsani